Papers with multilingual pretrained language models
FAD-X: Fusing Adapters for Cross-lingual Transfer to Low-Resource Languages (2022.aacl-short)
Copied to clipboard
| Challenge: | Adapter-based tuning is a technique that selectively updates language-specific parameters to adapt to a new language, rather than fine-tuning all shared weights. |
| Approach: | They propose to add light-weight adapters to multilingual pretrained language models (mPLMs) and add language-specific parameters to adapt to a new language. |
| Outcome: | The proposed adapter can enhance cross-lingual transfer from pretrained adapters for well-known named entity recognition and classification benchmarks. |
How to Translate Your Samples and Choose Your Shots? Analyzing Translate-train & Few-shot Cross-lingual Transfer (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have focused on zero-shot cross-lingual transfer of pretrained languages. |
| Approach: | They propose to use few-shot cross-lingual transfer to improve zero-shot performance of multilingual pretrained language models. |
| Outcome: | The proposed model can be scaled to high-quality samples and improves on zero-shot performance. |
TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data (2025.coling-main)
Copied to clipboard
| Challenge: | Existing mPLMs that handle non-transliterated data are not sufficient to train crosslingual models. |
| Approach: | They propose a framework to transliterate related languages into a common script by exploiting existing mPLMs and their tokenizer without any training. |
| Outcome: | The proposed framework can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training. |
TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | The world’s more than 7000 languages are written in at least 293 scripts, which poses a difficulty for multilingual pretrained language models in learning crosslingual knowledge through lexical overlap. |
| Approach: | They propose a framework that optimizes the Transliteration Contrastive Modeling objective to fine-tune an mPLM by contrasting sentences in its training data and transliterations in a unified script. |
| Outcome: | The proposed model outperforms Glot500-m on zero-shot crosslingual transfer tasks while retaining uniformity across scripts. |
The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing instruction tuned large language models (LLMs) struggle to understand cross-lingual sociopragmatic meaning (SM) lack of comprehensive investigation into their ability to understand SM is partly due to SM not being adequately represented in any of the existing benchmarks. |
| Approach: | They evaluate the performance of instruction tuned large language models (LLMs) on a multilingual benchmark specifically designed for SM understanding. |
| Outcome: | The proposed benchmark outperforms instruction tuned large language models on a wide range of tasks but falls behind task-specific finetuned models. |
ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. |
| Approach: | They propose a prompt-based method for token-level sequence labeling tasks . they propose to decompose an input sentence into single tokens and apply one prompt template to each token. |
| Outcome: | The proposed method outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer . the method also attains state-of-the-art performance when employed with the mT5 model . |
Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies show that multilingual pretrained models can recall factual knowledge without additional fine-tuning. |
| Approach: | They propose two parameter-free language representation projection modules to transfer factual knowledge between English and non-English languages. |
| Outcome: | The proposed modules improve factual knowledge retrieval accuracy and transferability across diverse non-English languages. |
ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrieval (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to improve retrieval effectiveness in high-resource languages are limited due to the lack of high-quality labeled data in non-English languages. |
| Approach: | They propose a modular dense retrieval model that learns from the rich data of a single high-resource language and effectively zero-shot transfers to a wide array of languages. |
| Outcome: | The proposed model performs well against state-of-the-art multilingual retrieval models trained on more extensive datasets in various languages. |
ScriptMix: Mixing Scripts for Low-resource Language Parsing (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing work has considered transliteration and vocabulary augmentation, but the consideration of combining the two has been lacking. |
| Approach: | They propose a multilingual pretrained language model that combines two strengths and overcomes the hurdle of combining them. |
| Outcome: | The proposed model improves POS accuracy by 14% and improves DEP LAS score by 5.6%. |
Multilingual Knowledge Graph Completion from Pretrained Language Models with Knowledge Constraints (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for multilingual knowledge graph completion do not align with mKGC tasks because of their English-centric bias. |
| Approach: | They propose to use multilingual pretrained language models to solve queries in different languages by reasoning a tail entity. |
| Outcome: | The proposed method outperforms the previous SOTA on Hits@1 and Hits @10 by 12.32% and 16.03% on public datasets. |
Multilingual Lottery Tickets to Pretrain Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Multilingual pretrained language models (mPLMs) have become the de-facto standard for multilingual tasks. |
| Approach: | They propose to search for a per-language subnetwork with comparable performance to the full model by scaling the model to reduce interference and then redistributing parameters to keep the parameters reduced. |
| Outcome: | The proposed model reduces the inference cost of models for each language while keeping the capacity per language more or less the same. |
Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to extend semantic parsing (SP) beyond English are challenging due to the complex slot alignment step after translation. |
| Approach: | They propose a method to enhance cross-lingual transfer for SP by utilizing mPLMs. |
| Outcome: | The proposed method synthesizes target language utterances from source meaning representations while maintaining high slot value alignment rates. |
Part-of-speech Tagging for Extremely Low-resource Indian Languages (2024.findings-acl)
Copied to clipboard
| Challenge: | Modern natural language processing systems thrive when given access to large datasets, but a large fraction of the world’s languages are not privy to such benefits due to sparse documentation and inadequate digital representation. |
| Approach: | They propose a parallel part-of-speech evaluation dataset for Angika, Magahi, Bhojpuri and Hindi. |
| Outcome: | The proposed approach improves F1 scores by up to 8% on Angika, Magahi, Bhojpuri and Hindi while ignoring the tokenization challenge. |
Zero-shot Cross-lingual Automated Essay Scoring (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to automate essay scoring (AES) use pre-trained multilingual representations and writing quality alignment to score essays in unseen languages. |
| Approach: | They propose a novel cross-lingual scoring method using pretrained multilingual representation and writing quality alignment to represent multilingual essays. |
| Outcome: | The proposed method achieves state-of-the-art cross-lingual scoring performance. |